To better support information retrieval tasks such as web search and open-domain question answering, growing effort is made to develop retrieval-oriented language models, e.g., RetroMAE and many others. Most of the existing works focus on improving the semantic representation capability for the contextualized embedding of the [CLS] token. However, recent study shows that the ordinary tokens besides [CLS] may provide extra information, which help to produce a better representation effect. As such, it's necessary to extend the current methods where all contextualized embeddings can be jointly pre-trained for the retrieval tasks. In this work, we propose a novel pre-training method called Duplex Masked Auto-Encoder, a.k.a. DupMAE. It is designed to improve the quality of semantic representation where all contextualized embeddings of the pre-trained model can be leveraged. It takes advantage of two complementary auto-encoding tasks: one reconstructs the input sentence on top of the [CLS] embedding; the other one predicts the bag-of-words feature of the input sentence based on the ordinary tokens' embeddings. The two tasks are jointly conducted to train a unified encoder, where the whole contextualized embeddings are aggregated in a compact way to produce the final semantic representation. DupMAE is simple but empirically competitive: it substantially improves the pre-trained model's representation capability and transferability, where superior retrieval performances can be achieved on popular benchmarks, like MS MARCO and BEIR.

On February 26, 1928, a headline in the New York Times announced, "MARCH OF THE MACHINE MAKES IDLE HANDS," with the subhead: "Prevalence of Unemployment With Greatly Increased Industrial Output Points to the Influence of Labor-Saving Devices as an Underlying Cause." What these alarming words referred to was the abundance of goods being produced in the roaring plants, mills and farm fields of 1920s America. According to a variety of statistics cited and charted by the Times, what Americans could now make was beginning to outstrip what they could consume, to the point of diminishing employment. "More and more the finger of suspicion points to the machine," the Times reporter, Evan Clark, claimed. "It begins to look as if machines had come into conflict with men--as if the onward march of machines into every corner of our industrial life had driven men out of the factory and into the ranks of the unemployed."